The ALICE analysis train system

In the ALICE experiment hundreds of users are analyzing big datasets on a Grid system. High throughput and short turnaround times are achieved by a centralized system called the LEGO trains. This system combines analysis from different users in so-called analysis trains which are then executed within the same Grid jobs thereby reducing the number of times the data needs to be read from the storage systems. The centralized trains improve the performance, the usability for users and the bookkeeping in comparison to single user analysis. The train system builds upon the already existing ALICE tools, i.e. the analysis framework as well as the Grid submission and monitoring infrastructure. The entry point to the train system is a web interface which is used to configure the analysis and the desired datasets as well as to test and submit the train. Several measures have been implemented to reduce the time a train needs to finish and to increase the CPU efficiency. 1. Introduction The ALICE collaboration is recording around 10 PBs every year. This amount of data is stored on different storage elements all over the world. To analyze a single dataset, on the order of 10 5 files have to be read where those files containing data on the order of 50 TB. The creation and submission of the analysis jobs to the Grid can be done with the ALICE analysis framework [1]. These jobs can be monitored and resubmitted using a monitoring service (MonALISA [2]). On top of these existing frameworks the LEGO (Lightweight Environment for Grid Operators) train system was created. It was developed to increase the CPU efficiency of the analysis jobs and thus to get more user analysis done with the same amount of computing resources. This is done by collecting multiple analysis tasks which should analyze the same dataset and by running them within the same analysis job. Following this procedure the data is read once, and then it is used for multiple analysis tasks. In comparison to that if the different user analysis would be done individually the data would be read for each analysis separately. This way more time would be used for loading the data and the CPU efficiency would be lower. Another advantage of the LEGO trains is to hide the Grid complexity from the users. They just have to define their code on a web page which also provides the …